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ABSTRACT 

Indicator checklists (questionnaires or checklists 
that contain selected predictive items from longer, comprehensive 
instruments) are used by states to monitor the conformance of human 
service providers with state requirements. This guide provides 
instructions for developing such a checklist as part of an 
instrument-based program monitoring system. The resulting instrument 
is intended for use by managers and technical specialists responsible 
for monitoring systems. It is pointed out that, although the guide 
focuses on monitoring systems for child day care., indicator 
checklists are potentially useful in other human service areas as 
well (e.g., foster care). The first section of the guide provides a 
brief introduction. The second section describes the origin of the 
indicator checklist concept as part of instrument-based monitoring 
and discusses the advantages of developing a checklist. The third 
section provides step-by-step instructions for constructing an 
indicator checklist based on an existing comprehensive instrument. 
The fourth section suggests ways of integrating the use of indicator 
checklists with comprehensive instruments and provides ideas on 
implementation. Appendices contain detailed information on technical 
issues such as quantitative methods for selecting particular 
predictor items. (RH) 
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I. INTROD! CT?0\ 

This Guide provides instnictiofis for ctevelc^ing 
an indicates cteiJclist as part of an instrunmit-taMd 
prc^ram mcmitOTing system* It is interuM for u% by 
managers and techn^ qiecmtists who are respcm- 
sibte for managing ami imiHOving monitcmng 
systen^ fc^ human serves. Though tte Guide 
focuses on mcHutoring systems fof child day care, the 
cc^Kvpt of an indicatm^ dmrklist has pcMential 
relevance for other sovkes (e.g., foster care) for 
whkh states have monitoring responsibilitMs. 

T)^ Gukle is organized into three secticN^: 

• What is an indicator ctecUst? This sep 
tion (tescribcs the <»igiu of the imlkatm^ 
checklist cw^cept u part of in^un^t- 
based monitoring and discuss the ^van- 
tages of (tei^lc^ing a cteck&t 

• How to tev^ip an faultoi^ ^eUM. 
This »cti<m provicfes a stq>4)y*'^ep gukte 
for ccnstnicting an imlk^OT ctocklist that 
b based on an existii^ ccmii^ehasive 
instnimrat. 

• How to im^^mit tte iMtfeatmr^ddist* 
This sectiim smggeste ways for integrating 
tl^ use of imikatcs' ctocktists with 
preliensive in^runmits ami pmwks ideas 
on how to imptement tte d^ddist* 

The appendices to the Guide contain detaifed infOT- 
mation on technical issues, such as quantitative 
n^ods for sdecting i»uticular proiictOT items, 
whkh sbcMiM be consickrtd in the process of ctevei* 
oping an indicatOT checklist. 



iL WHAT IS A\ ismc : 
CH^:cI::..^sT? 

An indicator checklist is a questionnaire or 
checklist that contains selected predictive items from 
a longer, comprehensive instninmu that a state uses 
to monitor service providers' conformance to state 
requirements. The state requirements may include 
health and safety regulations, fiscal compliance pro- 
visions in service contracts, or programmatic re- 
quirements that are the basis for the provicter's 
receiving public funding. 

The indicator checklist is a shortened venion of 
the comprel^nsive instrument. It contains item that 
have been determined to be most effectwe in discrim- 
inating between providers that typtcally receive high 
overall scores on the c(Mnprehensiw instrument and 
providers that typically receive low ovemll scores. 
Because these items distinguish wdl between pro- 
viders who are in strong compUance and those that 
are in wrak non«»iq}liaQce, they have been called 
"predictor" items. That fe, an individual who era- 
ducts a monitoring review shmild be able to use 
scores <»i ihc indioitor checklist to "predict" 
whether a given provider would have scored well or 
poorly on the comprehensive instrument. 

Background of the Indicator Checklist 

The concept of the indicator checklist grew out of 
instrumoit-based program mtmitoring (IPM) sys- 
tems. IPM systems arc monitOTing systons that rely 
on the use of objective, specific questionnaires or 
checklists, ratter than narrative site visit reports, as 
the basis for coraiucting reviews to oisuie that 
human services providers comply with a state's 
requirements. 

The advantages of an IPM system over narrative- 
based systems include: 

• consistency among momtors in the scope 
and depth of monitoring reviews; 

• assurance that all regulatory areas will be 
covered in suffidnit detail; 

• establishment of clear, objective expecta- 
tions of providers and state monitors; 

• simplined procedures for monitoring; 

• potential for cost efficiencies in the state's 
monitoring effort; and 



• promotion of quality programs. 

These advantages and other details of IPM systems 
are discussed at greater length in Volume 1 of this 
series. Guide for Policymakers. 

Once an IPM system is developed, the next logi- 
cal question is whether the comprehensive version of 
a state's questionnaire or checklist needs to be ad- 
ministered for every review of ail providers, even 
those who have maintained consistently high levels 
of compliance over the years. That is, would it be 
possible to eliminate cemxn questionnaire items for 
particular providers who have good recortfa of 
ccnniriiance withmit sacrifldng the quality of xUnt 
monitoring effort or reducing reguktory require- 
ments below desired, appropriate Icveb? The indi- 
cator checklist was developed in response to this 
questicm. 

Advantages of the Indicator Checklist 

The advantages of an indicator checklist that can 
be used selectively in place of the comprehensive in- 
strument are readily apparent: 

• The quality of the monitoring effort is 
nrmintained. There is no reduction in the 
need for all provide to comply with ail 
state requirements, and the state still has 
an objective, consistent basis for de- 
termining whether providers are in 
compliance. 

• The burden of undergoing a comprehen- 
sive review is reduced substantially lot 
ITOviders who have a history of high 
compliance and quality service* 

• The state has Iht potential for substantial 
cost reductions in such areas as staffing 
and information processing. 

• The state can reallocate staff resources 
that would have bwn applied to admin- 
istering the comprehensive instrument to 
other areas such as providing technical 
assistance to problem providei^ and tak- 
ing more effective action against providers 
who refuse to comply. 

• Reviews of providers may be con- 
solidated. For example, state staff ivho 
perform fiscal/contract compliance 
audits of providers might be trained to 
administer the indicator checklist during 
their audit. 



These advantages arc considerable for states that are 
currently' under pressure to c^)erate rowe efficiently 
and with the minimum necessary intcrfer^tce into 
the operaticms of private sector organizations* 

Concerns About the Indicator Checklist 

States may find tl^ advantages of the indicator 
checklist appealing, but some states wiU have specific 
Comoro that must be address^ before a decision is 
made to proceed with deveIopm«iL Such concerns 
wcHiId typically include the fdlowing* 

1. Caa evtry stale beMfit from an i^lcator 
dseckHst? 

Practically every state tfuit now has some form of 
questionmiife or checklist which contains weighted 
items can poientkUfy profit from using a shortened 
form of the instrument* Naturally, if a state's instru- 
ment is already sufficiently short, then little will be 
gaiittd by being even more selective abcnit questions 
or items to inclutte. However, many ^tes are con- 
fronted with lengthy instruments that cover a wide 
rani^ of requirements* Tt^se states aie prime can* 
didates for indicator checklists. 

On the other haml, if a state does not currently 
have an instniment-tesed syston, tten comi(feration 
of an indicator checklist is prmutture. In that in- 
stance, the state may be interested in reviewing other 
volumes in this scries on IPM systems that describe 
how IPM can improve monitoring and reduce costs. 

To develop a successful indicator checklist, it is 
imjK>rtant that the items on a state's current instni- 
mem be clearly linked to: 

• the state's requirements (e.g., regula- 
tions); and 

• the results or outcomes that are considered 
desirable with respect to the providers' 
performance in such areas as licensing, 
contract monitoring, and program 
quality. 

Unless there is a clear correspondence t^tween in- 
strument items aiKl requirements, there is a danger 
thai the items selected for inclusion on the indicator 
checklist may be perceived by providers as improper 
or illegal. 

Similarly, if items on a state's comprehensive in- 
strument are only weakly linked to the results that 
are expected from providers, then the grounds for 
selecting particular items as good predictors will not 
be solid enough. 



2 . How can the quality of monitoring N assured? 

Administrators may wonder whether the short- 
ened instrunient will compromise the quality of their 
state's current monitoring efforts* Our view is that 
the imlicator checklist will enhance current monitor- 
ing effcms by increasing tl^ efficient and effective 
utilization of monitoring staff; however, thm are 
I^ecautions that states shouU take in developing and 
using imlicator checklists. 

First, the short instrument should not be used as 
a sutetitute fm tl^ ccNnprehemive instrument, but 
rather as its complement. If the short form is viewed 
as tf^ monitcMring instnunoit, thai over time pro- 
viders may have a tcndemry to meet only ihe require- 
ments cov^ed on the indicate checklist. This 
c(Hild indeed compromise t)^ tevels of provide 
compliance. 

On the contrary, a state shcnild keep its com- 
pre)m»*ve instrument as the deflnidve set of com- 
pliance expectations and administer it for the initial 
review (e.g., licensing review) of a i^oWder. The 
state ctKikl sub^uently use the iiKlicator checklist as: 

• a scree^ag ikvice to determine wlKther, 
for a givOT provider, it is mcessary to ad- 
minister the l<mger version; and 

* an iatoim review instrument to be used as 
the principal t(K>l for providerj who have a 
good record of compliance. 

For example, the longer version would continue 
to be used both for '"problem" providers and on a 
periodic basis, perhaps evoy three years, for good 
j^ovidm. Naturally, if the indicator checklist iwe 
used with a provider and problems were discovoed, 
then the longer version, or some portions of it, could 
be administered. 

Over time, as conditions change, it will be 
nctxssary to ut^te and revise both the comprehen- 
sive instrument and the indicator checklist. Using the 
comprehensive instrument at least periodically with 
all provid«^ will establish a basis for modifying the 
shortened version to reflect changing compliance 
patterns. 

&cond, it is expected that both versions of the 
instrument would be used by state staff who are 
trained and competent to assess compliance (not 
necessarily only licensing staff or staff with formal 
training in child development). These staff would 
wrtainly not limit themselves to using the short form 
if they determined, on site, that conditions warranted 
using the comprehensive instrument. 



The purpose of the indicator checklist is to 
increase the options available to the state for moni- 
toring in a flexible and cost-effective manner, not to 
put unreasonable constraints or "blinders" on moni- 
toring staff. 

3. What are the potenUaJ drawbacks? 

As with all innovations, the Introduction of an in- 
dicator checklist as the basis fa* routine monitoring 
In a state may create some problems. Because no 
state has yet made wide${»ead use of sw:h checklists, 
it is difficult to identify all of the concmis that may 
arise in practice. However, a few potential problems 
can be anticipated. 

First, some states' laws require that all jwoviders 
be reviewed every year in all requirement areas (e.g., 
licensing regulations). That is, the state may insist 
that an annual review take place for each pro- 
vider using the comprehensive form of a state's 
instrument. 

In such situations, the use of an indicator 
checklist may require a state to chan^ its current 
legal provisions coiKxming the frequeiK^ and scc^e 
of reviews. A strong basis for making such a change 
is the cost-effectiveness of the shorter form, in that it 
can substantially reduce monitcning costs without 
reducing the quality of the monitoring effort. 

Second, the state staff who are responsible for 
monitoring may resist the inuoduction of the short 
form. From their viewpoint, it may appear that the 
use of indicator checklists is a reduction in the im- 
portance of their professional roles and that the 
state's cost savings may take the form of fewer jobs 
for monitors. 

If this resistance occurs, a state may need to 
assure staff that the mdicator checklist is not in- 
tended to reduce either their profosional judgments 
or the scope of the monitoring function. As men- 
tioned earlier, the indicator checklist must be used to 
complement, not substitute for, the comprehensive 
instrument in order for the shortened form to have 
validity. If anything, the professional judgment of 
the monitoring staff may be called upon even more 
frequently as it becomes necessary to decide whether, 
in a particular case, the shon instrument will be suf- 
ficient to measure compliance with state re- 
quirements. Monitors must be persuaded that the 
short form is an aid that is designed to reduce the 
monitors' workload, while offering clear-cut review 
guidelines, for those providers with whom the short 
form is appropriate. 



The reduction in workload could potentially 
change the relationship between monitors and pro- 
vidcn ffOTi one of relation to one of active sup- 
port in improving the health and safety of the day 
care environment and encouraging child devel<^ 
ment. This clumge in the mtmitors' role croild enable 
the state to make even l^ter use of Uie current 
monitoring staffs knowledge and experience. 

With respect to costs and staff reduction, there is 
little question that substantial ckcreases in workload 
could also result in reduced staffing levels. However, 
before considmng cutbacks in staff, states shouU 
cmder reallocating staff lime diac is saved because 
of the indicator checklist. This time could be ear- 
marked for («hcr monitoring activities such as 
technical assistance and training for providers. 

4. Otiicr legal issues? 

Since the first publication of this volume, ques- 
tions have been raised about the legal status of the 
indicator checklist, such as: 

• Can a state use the results of adminis- 
tering the indicator checklist as a basis to 
issue a license? 

• Can a state licensing agency uke action 
against a service provider based only on 
the results of administering the indicator 
checklist? 

In the first case, at least one state has received an 
opinion from its le^i counsel that such a use of the 
indicator checklist may be supportable under its sta- 
tutes. 

In the second case, it should be noted that the 
indicator checklist was designed as a management 
tool and a productivity tool, not as an enforcement 
tool. Neither the results of an indicator checklist 
review nor a comprehensive review are a substitute 
for the investigation and evidence gathering 
required to support an administrative or legal action 
against a provider. 

Other questions about the appropriateness and 
fegaijty of the indicator checklist will surely arise. In 
ail uses, the only safe answer is to consult with your 
Slate s legal staff before implementing the indicator 
checklist to ascertain any areas of possible legal 
exposure. 
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This section describes a method of designing an 
uidicator checklist to assess compliance with rtquire- 
mcms in chiU day care. Befcm proceedings the reader 
may fivd it i^pful to review the iten^ selected by 
Penmyivania for its ii^licator checklist (Appemlix A). 

GenerPl Approach 

The general approach for constructing the instru* 
ment is presented scl^inaticaUy in Figu*x 1 and con- 
sists of the following steps: 

1. Begin with your state's comj^rhensive in- 
strument. In this geroraliiDed approach, it 
is assun^ that ycnir state has such an 
instrument and that it rantaim a suffi- 
ciently large number of items that greater 
efficiency is desirable. 

Each item on your state's comprehensive 
instrument should be assigned a score or 
weight that reflects its relative impor- 
tance. These weights are essential in order 
to ensure that the items sefected for the 
indicator checklist according to the 
method prescribed in this Guide are the 
items that are most important to the 
slate.* 



2. Obtain at least one-year's data on day 
care providers* compliance as m^ured 
by the comprehensive instrument. The 
scores will used to categorize providers 
as either "high-grcnip" (providers in 
strong compliance) or *Mow*group'* 
(weak or non-compliant) providers. Con- 
scquently, a state must have sufficient 
confidence in its comprehensive instru- 
ment that it ts comfortabte using scores as 
the basis for distinguishing t^tween high- 
and low-^tHip providers. 

3. Use tt^ fcmnula described in Appendix B 
to klentify the it^m frcmi the comprehen- 
sive instrument that are mc^t usrful in 
distinguishing between high-gri ur and 
low-group providers. These "predictor" 
items Income tl^ basic items included on 
the indicator checklist. 

4. Make certain that all **cssent^" items 
from tl^ compretoisive instnimoit are 
included on the imiicator checklist. 
Essential items cover comfrfiaiKe areas, 
such as those pertaining to health and 
safety, that are so important that non- 
compliance with them might be tl^ basis 



^Rroctfdures fm devck^ng item weights <»- scores are ckscribed 
bckyw for chose suites XiM have im^ ytt met this prerequisite for 
devek^ng an indicator checklist. 
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Figure 1: Constructing the Indicator Checklist 



for denying a license to a provider. The 
predictor items may include es^tial 
items. It is possible, howevor, that essen> 
tial items will not be among the items that 
distinguish between hi^ and low-group 
providers because practically every pro- 
vider usually meets at kast the^ mini- 
mum requireiTCnt$. It is expected that in 
most states e^oitml itons are few in 
number. They should be acMed to the 
indicates checklist if they have not al- 
ready t^en iiKhided as predicts items. In 
Penmylvania, an exami^ of an essential 
item is: **Are all heatcn, space heaters 
that are fixtures, hot water pipes, heated 
radiators, and otlw sources of Irat 
exceeding 110* F that arc accessible to 
childroi equip|M»i with wrai^ing, insu- 
lation, partitions, or scrams?** 

5* The predktOT items jAus the e^ntial 
items constitute the imikator delist. 

Prerequisites for Developing the Indicator 
Checklist 

The goioal description of the n^hod fw con- 
structing the indicator checklist demonstrates that 
the necessary prerequisites for such an instrument 
are: 

• a comprehensive instrument in whkh state 
administrators have a>nndei^; 

• items that are assigmd weights or scores 
indicating their relative importaiKe; ami 

• su^Mnent (i.e., at least (»^year's) seem 
data from ing the comprel^nsive instru- 
ment to distinguish between high-grcnip 
and iow-group providers. 

It is also necessary that the state make a commit- 
ment to develop an indicator checklist Stat» that 
are interested in the indicator checklist coi^ept but 
lack one or more imrequisites may consult cHim' vol- 
umes in this series for furtlmr inf^mation ab<Hit how 
to proceed with developing an IPM system m may 
arrange more specific consultation through the 
Children's Services Monitoring Transfer 
Consortium. 

Step-By-Step Instructions 

The remainder of this section contains detailed 
procedures for developing an indicator checklist. The 
procedures lut based on Pennsylvania's expmence 
in designing its version of tl^ short instrument. 



1. Ai^irabe Your Comprehensive Instrument and 
Wdght Items 

TT» first step is to review your current instrument 
to (tetermine that an indicator checklist wmitd be 
useful and assign wdghts to' items on the instrun^nt, 
if tl^ weights do nm alreiKiy exist. Tt^ procedures 
are di«nis^ in detail Mow. 

• Revfew lastranmBt 

An mstrumoit with more than 50 items provides 
a good bads fm de^c^g an tmlicator checklist. 
Even if there are fewer items, a state may find it 
useful to shOTten its instrument, especially if each 
quesdon or item is furtter subdivided. However, 
thm are rebtively fewer advantages to be gained by 
producing an imiicatm^ checklist based on already 
shmi insmunents. A brief description of Pom- 
sjdvania's comi^dmisive instrument may be useful 
as iM^kgrcHu^ tlw rest of the instructions and an 
examt^ of an instrumrat that b a good caiKiidate 
for devdoimient of an indicator checklist. 

P«msybania*$ comprehei»ive instrument^ con* 
taim 276 items that are grcmped into se^^ categ(»ies 
^xording to state regulaticms; (1) Administration, 
(2) Environmoital Safety, (3) Child Development, 
(4) Nutrition, (5) Sodal Servfces, (6) Transportation, 
and (7) H^th. A total score for ^h cat^wy aiKl a 
grami tcMal score are calculated using waghts that 
have bem assigxmi to es^ item. The weights ran^ 
from 2.80 to 48.00, based <m tl% degree of risk to the 
chiU in day care. Low wdghts indicate relatively low 
risk iten»; itrais with weights of 48.00 are considered 
such high risk items that non-compUaiKx with one of 
these "essentials'* is si.ffident to deny a Ucense in 
Pomsylvania. 

Sct>rcs on Pennsylvania's comprehensive instru- 
nmt may rai^ fnm zero (perfect cmipUance} to 
1447 (tcMal n(»i-COTiplia%e), with providm recdv- 
ing the sccm/wdght assigned to each item fcmml out 
of compIiai^« That is, low s^n$ indicate that pro- 
vi<krs are in high ccrnipUaim white high scores in- 
dicate inoviders are in low or non-compliance. 

• Assign Weights 

Questionnaire itons may be assigned weights 
only on the basis of scmie criterion or standard. 
Pennsylvania used the * 'degree of risk to children'' as 
the criterion fOT assigning weights to the it«ns in its 
comprel^nsive instrumem. The term "risk" in- 
cluded all tyj^ of risk, such as risk of physical harm 



*See VolufiM ! amiJc^fhss ser^sfor ifla^tnutons of Penn- 
sylvania^ comprehensive instrument iorwdi. 
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(e.g«, from unsafe facilities), risk of psycholc^ical 
harm (e.g., from callous and uncaring treatment), 
and risk of developmental harm (e.g., from not hav- 
ing some form of development program). 

Using degree of risk as the criterion, Penn- 
sylvania askoJ \00 individuals to rate each item on 
tl^ state's comprehensive instrument on a scale of 
zero (no risk) to S (high risk). The 100 indivkiuals 
represented a variety of penpectives, including pro- 
vides of day care, state human services staff, ai^ 
other child development professioiuds (e.g., unive- 
sity researchers). Tht results of t)^ survey were com- 
piled and averages for each item were calculated. 
Because the range of tl^ aveages was so rmrrow (.28 
to 4.90)^ very small differei^ (e.g., the difference 
between 2.8 ami 3*0) represented substantkl dif- 
ferences in pemived risk. Pennsylvania multiplied 
all item values by a factor of ten to produce a ranp 
of 2M to 48.00. States intending to comiuct a 
similar survey to establish weights arc advised to 
begin with a 10-point scale rather than the S-point 
scale used by Pennsylvania. 

The multiplied averages constitute the 
weighting/scoring system that Pennsyivank uses. As 
might he expected, the survey results indicated con* 
siderabie agreement among the respondents concern- 
ing which items involve risk to chiklren, and the fa^ 
validity of these results wb% sufficient for Penn- 
sylvania to adopt tl^ averages as the relevant weights 
without further statistical testing and reflnement. 

The importance of assigning weights to items on 
the comprehensive instrument is revealed in the types 
of questions that were selected for Pennsylvania's in- 
dicator checklist,* Practically all of the items on the 
indicator checklist are highly weight^, reflecting 
their importance with respect to the degree of risk for 
children. The likelihood that these questions would 
appear on the indicator checklist was enhanced 
because their high weights were crucial in placing a 
provider in either the high group or the low group (as 
described in Seep 3 below). If no weighting system 
had been established, then providers might have been 
placed in the high or low groups based on percentage 
scores of items in compliance. Using simple, 
unweighted percentages increases the possibility that 
compliance with administrative regulations such a^ 
posting menus for children's snacks would take on 
undue importance in differentiating high- and low- 
group providers and that items based on such regula- 
tions would be chosen as predictor items. 
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2. ObtaiB At Lent Ow- Year's Data/Scores on the 
Comimfl^fisive iiBtniment 

The availability of providers' scores on the com- 
prehensive instrument for a period of at least one 
year is a necessary prerequisite for development of 
an indicator checklist. In Pennsylvania, the com- 
prehensive instrun^nt has been used to license pro* 
viders since 1978, and the state has detailed provider 
data for three years. 

3. Use Foraula to Identify Go<nI Predkrtw Items 

The formula presoited in Apt^ndix B can be 
used to selea predictor items once c»iain prelimi- 
nary steps have \xcn taken to identify the grcHips of 
providers whose scores will be used in the formula. 
The bulteted sections Mow descril^ these $te{», as 
wdl as the tmsic logic involved in using the formula. 

• Identify ''High-Group*' and '*Low-Group** 
Providers 

The first step in selecting predictor items is to 
take a sample of providers for whom scores are avail- 
able and to identify subgr<ni{» of providers who have 
histories of good and bad compliance. These 
subgroups* scores are used in applying the formula. 

Pennsylvania selected a random sample of 200 
providers from the state*s roster of ISXK) licensed 
providers. This 20 percent sample was judged to be 
large enough to incluck most types of providers (e.g., 
profit, non-profit) and geographic r^ons (e.g., 
s<Mitheast Pennsylvania, central Pennsylvania). A 
samite of 200 also allowed quartile figures (50 pro- 
viders to a quartile) that were large enough to create 
confidence in the analytical results. 

Hie 200 providers in the sample were then listed, 
according to their remit scot& on the comprehen- 
sive instrument, from the provider in the highest 
compliance (lowest numerical score) to the provider 
in worst compliance (highest numerical score). 

The top 50 providers (highest quartile) were 
designated the high group, while the bottom 50 pro- 
viders (lowest quartile) were designated the low 
group. The middle group of 100 providers were ex- 
cluded from the rest of the analysis. 

• Tsbuiale Item Non-Compliance 

For each item on the comprehensive instrument, 
a frequency count was made of how many high- 
group providers were out of compliance with the 
item and how many low-group providers were out of 
compliance with the item. The results of this analysis 
were tabulated using a form like that in Figure 2. 
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Flgm 2; FivqwocT irf I|R» ObI of C 

• Sort wd Select Predictor Items 

The objective of the sorting ami sdccticm pro- 
cedure is to identify those items from the comprrherf- 
sive instrument that are useful fOT distinguKhing be- 
tween high- ami low-group providers. These items 
are designated predicts ftens. 

The logic of the sorting and selection procedure 
may be illustrated as follows. For every item on the 
tabulation list (Figure 2) it is possible to construct a 
table like that in Figure 3. 
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Ffgm 3: £su^ of Good Pivdiclof 
litm f 9 In Frtqwocy Tsbk (F^rr 2) 



Figure 3 illustrates a good predictor item. It 
shows that ail SO of the high-^oup providers were in 
compliance with the selected item (in this case Item 8 
from Figure 2) and that all 50 of the low-group pro- 
viders were out of cmnpitance. Cksriy, this item 
from the questionnaire is useful for making distim;- 
tions. Thus, if a monitor found a randomly selected 
provider to be in compliamx with this item, then 
there is a good probability that the provide would be 



among the high-group providen as measured by the 
comprel^nsive instrument. This item would be in- 
cluded in the indicator checklist. 

Figure 4 illustrates another possible situation. In 
this case, of tl^ high-grcmp providm was in 
compliance with the item white all of \hc low-group 
providers cMiplied. It is difrtcult to imagine a ques- 
ticmnaire or dm:klist itrai for whkh this ^tuaticm 
wouki occur, but it is a logical possibility. An item 
that reftected this patton ccHild not be as a 
positive predicts of a high-grcHip or low-grc^p j^o- 
gram. It would not be included in the indicator 
checklist. 
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Figures 5 and 6 illustrate other situations that 
could occur for panicular items. In Figure 3« all of 
the providers, both high-grmip ami low-group, are 
out of compliance on the item. This couki occur if 
the item is very difficult to comply with, too c(»tiy to 
correct, or <Kit of the provklers' control to correct. In 
any case, this type of item wchiU mn be a good fro- 
dictor fw distinguishing between high- and low- 
group providers. 

The opposite situation is illustrated in Figure 6, 
where all of the providers are shown to be in com- 
pliance with the item. One could easily expect this to 
occur if the item were very easy to comply with or if 
the item ref^uented a minimum neossary require- 
ment for operating, for example, a building that met 
fire safety code requir^ents. The iton shown in 
Figure 6 would not be a good predictor item for dis- 
tinguishing between high-group and low-group pro- 
videis. A state might, hiwevo'p want to indtKie 
certain items that are like this on its indkatCH* check- 
list if the items are esseattol iteim pertaining to the 
health and safety of children. These essential items 
woukI presumably be those that wouki place childrm 
at high risk and that would be sufficient grounds for 
denying a license to a provider who was (Hit of 
compliance. 
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Figures 3 through 6 ilhistrate situathms that may 
be atypical in actml {nwtke* rinoc ttey pcmray all 
50 of the high-gnmp ami Um^group provictes as in 
compliance en* wt of ccmiplianoe with partkuhu* 
items. In practice, it is more hkeiy that scmu: of each 
group would be in cmni^huice aiui otters in ci^ 
grouo w<nild be wt of oxnpiiance with a givoi iton 
(e.g*. Item No. 14 in Hgure 2). It a necesary, tten, 
to have a method fw nwamring gr^tkms ami cfe- 
grees of an item's usefulrass in disdi^uishing be^ 
tween hi^i-group and low-group |Mtyvktes« Ox 
such n^hod is to cakukte a c<»Tdaticm coefficfeni. 
Prc^urcs fm setectmg {mdkt(»r items u^i% a cor- 
r Ut on c^nicient (phi coef^km) are (kscribed in 
Appeiuiix B. The ccnrelatkm coeffkient simi;riy jm- 
vides a convenknt quantitative n^tod kfentify- 
ing whkh qimtJonnaire items are better and worse 
tor distinguishing between high- and low-group ihx>- 
viders using a log^ api^oach idoitkal to that il- 
lustrated in Figures 3 tlm^igh 6. 

4. Add Essential Items 

In general, a state might find it desirable to in- 
clude certain questionnaire items that are not good 
predictors but are noneti^less extxcn^ impOTtant 
for ensuring that emothd minimum rtquiremrats 
are met by all providers. The^ essential items are 
usually few in numbor and potain to l^u^ con- 
siderations of health and safety. Typically, all pro- 
viders that are licensed will conform because of this 
essential nature. For example, an q^cn hrat scmrce 
would be too hazardous to permit licensing. 

It is possible that some of these es^ntial items 
will also be gocKi predicts items and would he in- 
cluded on the indicator checklist Ixcause of their 
ability to distinguish between good and unacceptable 
programs. If, after looking at the items that are se- 
lected as predictors, a state decides that additional 
essential items must be imrluded on the imlicator 
checklist, these can readily be added. A state should 
use discrimination in adding these osential items to 
be sure that only the most important items are 



F^ptfv ic Esam^^ of lien Ttet b Too Eny 
ItaB 13 ia Fw qo ga c y IMt (F%m Z) 

added. Otterwise, the indicate diecklist ccniki ex* 
pami raiHdly ami titc ^vantages of the shortened 
instrun^t wou^ be lost 

BecauK essential itens on PeansyNrania*s com- 
imlimisive instrun^t were so highly weighted, 
many of tten were sdected as good imdictw items, 
aiKi it was noi necessaiy to ttem separately to 
the indk^tcn* ctedd^ as es^tial items. 

5. Cossfrvct the Indicator Cteckfist 

The comfikit ImlkatOT dieddist will contain 
b(Mh imdtct(»- itom ami e^rotul items. Dqxmding 
on tte standards ^ by the state in selecting essential 
itms, tte indkatOT delist shimU be a much m«e 
versatile instrunmit than the cwnprehensive instru- 
n»it ami shi^ild require far kss time to administer. 
In Pamsylvania, tte ccmi|mhensive instrument, 
which ccmtained 276 itons, was the basis for an in- 
dicatcM' d^ckfist that required cmly 18 items that 
were selected using the procedures described above. 

Several refinemoits to tl^ imlicator checklist are 
pmsibte dq>miing on the state's policies ami tt^ 
resources available for developing and maintaining 
the state's IPM system. 

First, the state may reHne its indicator chedclist 
by ^gregating tht i»^tctor iteim by area of regula- 
tion, for example, the sevai areas of regulation 
covered in Pennsylvania's comprehensive instru- 
ment. Th^ procedure descritoi above for selecting 
predictor items did not involve a conscious attempt 
to include predictor items from each area on t}» in- 
dicatCK- checklist, but focused instead on the general 
predictive value of the items to be included on the In- 
di(^tor checklist. 

States may select area-specific items simply by 
generating a different list of high-group and low- 
group providers on the basis of scores for each area 
of the regulations and tt:^ following the procedures 
for tabulating item responses and selecting predictor 
items. For example, a state would list providers in 
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order of. their recent scores in the area of en- 
vironmental safety and then follow the procedures 
for distinguishing the high and low groups and 
selecting the predictor items. The state would repeat 
these ste{» based on scores for transportation, ad- 
ministration, child development, and so fortli* The 
ranking lists generated for each area would be ex- 
pected to show some differences in the providers 
selected for the high group and the low group. 

Second, states may consider revising their com- 
prehensive instrument scoring to account for partial 
compliance with particular items and tt^ revise (he 
indicator checklist on the basis of liwx revised 
scores. For example, providers would be given credit 
for being in SO p^cent compliance with a particular 
item rather than being marked out of compliam:e for 
not meeting the item's standard 100 percent. 

While these kinds of refinements are appealing 
conceptually, it is difficult to tell how diff^ent the 
indicator checklist would be as a result of the 
changes without actually testing the refinements in 
practice. To our knowledge no state has yet per- 
formed such tests. 

Maintenance of the Indicator Checklbt 

Items on the indicator checklist wf' v id to be 
reviewed and changed periodically* Kcv .Ions are 
needed to: (1) reflect changes over time in the com- 
pliance of providers; and (2) prevent providers from 
regarding one version of the checklist as the only set 
of state requirements. 

For example, it is likdy that regulations wiU 
change over time and that providers' compliance 
with particular items will change as their clientele, 
management, staff, and even physiod facilities 
change. The clwcklist should be revised in response 
to this evolution. A state should plan on revising its 
checklist at l^t every three yrars, following the pro- 
cedures described above. 

An alternative appr(^ch would be for the state to 
develop three different checklists (e.g., a new one 
each year for the first three yrars) and then to ad- 
minister the three instruments to providers on an 
alternating basis for three or four years. Having 
more than one version of the shoner instrument 
gives the state greater flexibility in assessing pro- 
viders* compliance with a broader range of checklist 
ilems. 



Each time the state prepares a version of its in- 
dicator checklist, it will need ^res from the com- 
prehensive instrument for providers in both high and 
low compliance. To ensure an adequate number of 
strong providers' scores on the comprehensive 
instrument, the state should consider using the com- 
prehensive instrument to monitor some low percen* 
tage (e.g., 10 percent) of the strong providers on a 
random basis each year. 
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The su^cssful implementation of the indicator 
checklist will depend heavily on three main factors: 

• confidence in the indicates checklist; 

• dear expectations on the pan of top state 
admiimtratOTs, monitoring staff, and pro* 
vidm aixMt what the indicator checklist is 
intended to accomplish; ami 

• organization of the implementation 
effort. 

Each of these factors is discussal below. 

Confidence in the Indicator Checklist 

The state mvat be ccmvinced that the indicator 
checklist will ensure that i^oviders stiU ccmii^y with 
all state regulatiims. In Opting from the craipre- 
h«isive instrument to the short form, tnat is always 
a possibility that providers will gradually cease to 
comity with the items that are excluded from the 
sh<Ht form. Top admini^rators must be ccmvinced 
that the shortoied instrument is valid ami c^»ble of 
distinguishing between providers whose compliance 
is high and those who are not in compliance. 

The state can foster this confldmce by: 

• developing an indicator checklist that has 
face validity, covers the critical areas to l» 
enforced, and contains predictor items 
that are good indicators of high and low 
compliance; and 

• testing the indicator checklist to make sure 
that it will yield useful results. 

An instrument is said to have **facc validity*' if 
an experienced practitioner in the omipation for 
which the instrument is designed judges the instru- 
ment to be accurate and appropriate. This under- 
scores the importance of prof^onal opinion and 
professionals' ability to weigh numerous subjective 
factors that statistical tests can only approximate. To 
check the face validity of the indicator checklist, the 
state could circulate the newly designed instnunent to 
state monitoring staff (espraally top adminis- 
trators), independent researchers such as university 
faculty, and perhaps a cross section of the day care 
providers. The reactions of these reviewers can be in- 
valuable in detecting problems with predictor itrais 



and also in sugg^ting additional essential items that 
should be induded r^ardlcss of their predictive 
value. 

Tt^ Mcond m^od for building confidence is to 
pilot test the indicator checklist. This can be done in 
two ways. The last apoisive way is to conduct a 
statistical test of \hc instrument u»ng historical data 
that are already in the mat's files. It is reai»>nable to 
cx^ct that if the itons cm the shcntened form had 
beat used in the in jAs^ of the cmnjH^dioisive 
f(mn, the state wcmld have re^tmi similar condu- 
»ons concerning ccmipliann. 

To ccMKhict this test, the state simply sdects a 
randcrai $am{^ of i^ovklo* evaluations that were 
ctmq)leted in tte past u^ng the ccmiprehensive 
u^tnumnt The state then calculates tte scores that 
the inx)vid€T5 wouid have received if the short form 
Imd beoi used, (Rammbor tte d^cklist questions 
are a sul^ of tt^ questicms on tte omiimtoKive 
instrument) Then, the results of using the short 
form versus the icmg^ version are compared. If both 
fmrns wcniki tuve rented in similar ciM^hisiom 
ctmceming apprcqmate actions to take, then the state 
can have greater confidoice in the shcniened instru- 
ment, Penreylvania k currently conducting tl^sc 
experimoital tests of its indkat<^ checklist. 

A secoml way to test the imiicator checklist is to 
use it on a pilot basis with providers in current 
monitoring situations. This kind of test can build 
confidoice that tte right items have bem iiKlucfed on 
the shortened instrunmit. It can abo yield necessary 
infcmnation on how providers will respond, what 
tyi^ of discretion must be used by the monitors in 
deciding whette- the indicator checklist should be 
used with a {articular provider, and how use of the 
indicator checklist ran increase effiderKy. 

The state should ccHKluct both types of tests 
before full-scale introduction of the shcmered instru- 
ment. On the basis of these tests the instrument can 
be improved to yield even more accurate findings. 

Clear Expectations About the Use of the 
Indicator Checklist 

The indicator checklist is designed to be used to 
complement, not replace, the comprehensive instru- 
ment. The uses of the shortened form as a screening 
device and as an interim monitoring tool for good 
providers have been mentioned earlier in this Guide. 

Top administrators for a state must establish ex- 
actly how the indicator checklist is to be used and 
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communicate their decision to both the state's moni- 
toring Stan* and providers. Polides concerning the 
monitc^' range of discretion in using the short and 
comprehensive versions of the instrument should be 
specified so that monitors know clearly tte extent 
and limits of their authority in particular situations.* 

Finally, top administrators must reo^nize that 
the move to an indicator checklist inevitably means a 
reducticm in tl^ amount and detail of comidiat^ 
informaticHi al^nit individual providers. Hie short 
form dearly generates greato* uncertainty with 
respect to providers' ccmirfiance with all r^ulations, 
and it is p<^ble that instaiKxs of non-com|4iaiKe 
will be overlooked by monitms with suiverse con- 
sequetxxs for chiUrcn and f<K the state. A weQ 
designed indicates clmklist can ovmtmie thcx 
potential probtems to a large extent Before com- 
mitting the state to using the indicator cbeck&t, tc^ 
administrates shmild, howevn, explicitly wd^ tlm 
trade-offs between the benefits of cost savings, re- 
duced burdens on providers, and more elective 
allocation of state resources against tte costs of in- 
creased uncertamty. 

Organization of the Implementation Effort 

The successful implementation of any dian^ in a 
state's system will depend on having a clear, well 
organized implementation plan that ii^tudes the 
following: 

•Cfear objectives that specify what is to be 
accomplished, why the state is devdoping 
its indicator checkUst, ami what issues are 
likely to arise that wiB influom the 
development effort* A state can use this 
Guick and other publi^ons in this series 
to clarify these issi^. 



•A ^tdufe for imi^ement^ion that shows 
all of tti« tasks to be accompUshed, their 
quotes fw Qsmfkiikm^ critkal c€Mnpletk)n 
dates, ami the timing of i^ogxtss reports. 
• 

•A bud^ that covers the allocation of staff 
time aiKl other resour^ to particular im- 
pten^tation tasks. Otho' resources may 
indiMie fumis fa printing and computo* 
time* 

When the fflm has been drafted, it should be re- 
viewed and ai^>roved by all state oHicials who have 
contrd over tte work to be done and the romirces to 
acccmipUsh the tasks. The plan shcmki be reviewed 
pmodkalty in the course of tmi^mentatim, and 
necessary reviaons ^Kmkl be macte and agreed upon 
by all managers axui ^aff invc^ved. 

An iOustrative wc^k |^ impkmenting a 
short-fcmn in^numnt is presented in Figure 7. Tte 
illustrative plan shows each of the steps described in 
this secticm in additicm to the stei» invcrfved in the ac* 
tual design of the instrument. The assumes that 
the state already has a c omi we h qisive instrument 
with weighted items and that at least one-year's data 
are available. 



*See Appendix C fori descnptioti of policy of^ions. 



•Ctear assignrarals of spedfic respon- 
sibilities to the individual staff members 
who will devdop and implonent the indi- 
cator checklist. These staff will typically in- 
clude the head of the state's licen^ng or 
monitoring unit, indivkluals with knowl- 
^ge of research and evaluation methods, 
selected supervisors in the state's field of- 
fices, and management information 
systems staff. Care should also be taken to 
include assistance from the stated legal 
staff in developing and reviewing the 
checklist. 
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APPENDIX A: PENNSYLVANIA'S LIST OF PREDICTORS 



A list of Pennsylvania's predictor items is reproduced in its entirety in this app«idix to iUustraic the Idnds of ques- 
tions that can be selected using the method described in this Guide. 

The questions are grouped according to the seven regulatory areas covered m Pennsylvania's comprehensive 
instrument. The first numbo- in parenthesis after the item heading is the item's number on the comprehensive instru- 
ment. The second number in parenthesis is the weight that is assigned to the item for purposes of scoring. 



ADMINISTRATION (A) 

A-1 Health appraisal (#37) (13.50) 

An staff, ii^uding tcmpcH^ am! substitute 
employees and volunteers who serve on a regular 
basis, who come into contact with the children, or 
who work with food i»q»ration, have a iMalth ajh 
praisal within 3 months i^or to providmg intitial day 
care service ami annually ttoeafta-7 (Health ap- 
IH^aisals shall be cotifwd by a licensed physkaan.) 

A-2 Physical exam (#133c) and Immunizations 
(#133e-i) (4.39 each item) 

From the sample selected does each child have an 
age-appropriate health appraisal on record which in- 
cludes the following: 

(c) physical examination? 

(e) required number of DPT immunizations 
for child's age? 

(f) required number of polio immunizations 
for child's age? 

(g) required measles immunizations for 
child's age? 

(h) required rubella immunizations for 
child's age? 

(i) required mumps immunizations for 
child's age? 

ENVIRONMENTAL SAFETY (ES) 

ES-1 Hazard Free (#48) (45.10) 

Are play areas free of hazards and unsafe areas such 
as open drainage ditches, wells, holes, and heavy 
street traffic or surrounded by fencw or natural bar- 
riers to limit access of children to hazards? 



ES-2 Cfeasisg materials/ deiefigeiil KcesdbUity 
(#64) (47.10) 

Are an ekaning materials, d^ergmts, am>sol cans 
and (Mhtr {x^mous and toxk material kept in a 
place imwccssibic to chiklrai and separate from child 
care areas, food ami food preparation areas? 

ES^ Miag imiat/holcs (#66b) (21.40) 

Are all room surfaces 0nduding ceilings) free of 
peeling paint or plasty, cracks and holes? 

tS-4 Inaccessible beat source (*75) (48.00) 

Are all beaten, si^ce heatm that are Axtures, hot 
water pipes, heated radiators, and other source of 
heat exceeding 1 10 degrees F accessible to children 
equipped with wrapping, insulation, partitions or 
smens? 

ES-5 OutsMe phcme Uiie (#77) (22.05) 

Does the facility have a telephone number which is 
listed? 

ES-6 Outdoor ^uipmesl (#82) (45.80) 

Are all outdoor toys, play «juipment, and all other 
equipment used by the children of substantial con- 
struction; in good repair; and free from rough edges, 
sharp comers, pinch and crush points, splinters and 
exposed bolts, and unguarded ladders on slides. 

ES-7 Water safety instruction (#87) (45.80) 

Have all staff, volunteers, and other adults who are 
counted in the adult-child ratio for swimming re- 
ceived basic water safety instructions from a person 
certified in Hfesaving? 
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CHILD DEVELOPMENT (CD) 

CD-I Observattons (#99) (9.00 each item) 

Do aciivities promote: 

(a) development of skills? 

(b) sdf esteon? 

(c) positive self-identity? 

(d) choice of activities? 

(See COFAS, opposite page.) 

CD-2 Sapefvistofl of chUdren (#102) (4S.60) 

During the site visit, did staff supervise the children 
at aU times, both indcK>rs and out? 

SOCLiL SERVICES (SS) 

SS-1 AppUcatioB ffonn (#154) (3.69 each item) 

From the sample selected, does each child's applica- 
tion include the following: 

(a) name and birthdate of child? 

(b) any special disability? 

(c) full naoMS, home and wwk addresses 
and home and work tdephone numbCTS 
of i»rents? 

(d) name, address and telephone number of 
child's physician or source of medical 
care? 

(e) name, address, telephone number of 
emergency contact person? 

(f) health insurance ojvera^ for the child 
under family insurance policy or 
medical assistance? 

(g) signature of |»rents? 

SS-2 Agrwmenl Form (#155) (4.06 each item) 

From the sample selected does the agreement for day 
care service include the following: 

(a) amount of fee to be charged per day/ 
week? 

(b) date fees will paid? 

(c) services to be provided by the provider 
which are part of the fee and those ser- 
vices which are extra costs? 

(d) hours of care including arrival and 
departure times? 

(c) person designated by parent to whom 

ihe child may be released? 
(f) date of admission? 



(g) signatures of administrator or director 
and the parents? 

TRANSPORTATION (T) 

T-l Safety carrier (#126) (45.50) 

Is each vdiide equipped with operable age- 
ai^MTOiNlate safety carnors or restraints for each 
child? 

T-2 Vehicle la raotkm (#127) (45.50) 

Are ail children seated and secured in restraints at all 
times the vdiidte is in motion? 

HEALTH (H) 

H-1 EmefftBcy Contact (#136) (12.96 each item) 

From the sampte select^ is there emergency contar 
inftmnation on e»:h child induding the following; 

(a) nan», widress and tdephone number of 
child's i^ysidan or source of Iwalth care ' 

(b) h<»ne and work addresses and home and 
vnxk tetei^ioRe numbers of both parenu ' 

(c) name, address and telephone number c 
cmCTgoicy contact person? 

H-2 Adminbtration of Medkalion (#140) 
each item) 

Does the facility have any children receiving n: 
tion or health aids (i.e., cough drops, a 
vitamins, ear drops, aspergum, cough syrup. • 
ide, ata^d, ointmoits, creams, mercuroc;. 
iodine, methiolate)? If yes, do the records/ rr. 
tion lc« indude: 

(a) physician's curroit written insirucuo- 
f<a all prescriptitm mediation? 

(b) iMtfoit's current written instruction 
ail non-prescri)Hion medication? 

(c) written consent from child's parents fi 
prescription and non-pre$criptio; 
medicaticHi? 

(d) rea>rd of dose and time medication i 
administmd? 
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CAREGIVER OBSERVATION FORM AND SCALE (COFAS) 



ThB form is designed to be used by the monitor in recording specific ol»ervations of caregivn^' child development ac- 
tivitks during a 30-minute period. See Volume 3 of this «ries for detail^ instructions for competing and scoring this 
fonn. 

No. of children preset at the beginning of the care^m observation 

No. of adults pment at the b^inning of a car^ver observation 

Time of (4)scrvati(Mi .. Nan» of caregiver 



During your (Hiservation did ibe caregivn^ Two-minute Ot»ervation Periods 









1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


Weights 




Speak unsolicited to a child 


01 




















1 




Use the child's dialect 


02 




















1 -M 


Hi 


Reqxmd voliaUy to a cfaU's speech 


03 






















"«-2 


O 
< 


Read i« kfentify lectures to a diiM 


04 






















-M 


O 


Sng or play nnisk with a duid 


OS 
























z 


Spealc stowly and ^arty to a chiU at all times 


06 




















1 


3 


ItuemqM or cut off a child's verbattzatim 


07 




















1 -3 




Scream or yell at cbiUren 


08 




















1 -30 




/%in#w iHwc ic^^ lo oecoroc so nign u is nara 
f<^ observer to umlerstaitti childrai 


09 






















i — i 




Give affectii^te iMiy^al conuct to chiU 


10 






















+ 3 




iViaKc wuvicy sugi^sufju lo a cniMi 


ii 






















■H 




Pfay^cally punish a chiU 


12 






















-100 


< 
Z 


Use food as reinfmcement 


13 






















-3 


o 


Make ftin of or ridkuk a child 


14 






















-30 


XriO-EMOT 


Lei (Hter chiMren make ftm of en ridicufe a 
child 


15 






















-30 


Verbally criticize, sc<^ w threaten a child 


16 






















-30 


Isolate a chikl i^iysically 


17 






















-1 




Ignore a child's request 


18 






















-5 




Interrupt a child*s activity kiKi prevent 
its rampietion 


19 






















-5 




Leave the child alone 


20 






















-40 


MOTOR 


F(^ter development of child's gross 
motw skills 


21 






















+ I 




Show impatknce or annoyance with child's 
questions 


22 






















-2 


IITIVl 


Use to^ms which are above a child's 
reasoning ability 


23 






















-3 


§ 


Deal in abstract concepts without 
concrete exampks 


24 






















-3 




Show intolerant with a child's mistakes and 
not accepting faulty thinking 


25 






















-2 




Prepare or serve food for a child 


26 




















1 ^ 


Prepare activities ot arrange the room 


27 




















1 ^ 




Do nothing 


28 




















I ~ 




Talk with other adults 


29 






















Record the type of ftctivsiy: (e.g., listening to 
records, reading a story, an project, etc.) 
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APPENDIX B: SELECTION OF PREDICTOR ITEMS 
USING THE CORRELATION COEFHCIENT 



The corrdation (phi) coefficient is a measure of how weU a givtn item on the state's comprehensive instnuncnt wiU 
predict whether a provider faUs into the high-c<MnpliaiKX or kw-compUance group. The phi coeffidoit is computed 
arithmetically and can be used to select predictor items fw an indicator dMcUist. The pW 

similar to that used in gerorating Figures 3 through 6 in the text. To calculate a phi coeffi^ it is first necessary to 
produce a frequency table such as the table in Figure 2 f or aU items in the state's comprehensive mstrument. 

The formula for the phi coefficient is: 
, ^ (A)(P) - (B)(Q 

^ ^(X)(Y)(Z) ;j 
where the letters in the formula rej^esent the numbers in the corresponding cells in the matrix: 





Providers 
fa CoMl^Bfe 


Providefs 
Oat of Cmai^aKe 


Row 
Total 


High 
Group 


A 


B 


Y 


Low 
Group 


C 


D 


Z 


Column 
Total 


W 


X 





The foUowing ranges of predictor values obtained by using the phi cocfncient formula can be used to select items 
for the indicator checklist: 



^ = (Rflitge) 

(^1.00) -( + .26) 
(-^.25)-(0) 

{0)-(-.25) 
(-.26) -(-1.00) 



C&uvcterbtk 
of Item 

Good predictor 
Too easy 
Too difficuh * 
Terrible predictor 



Dec^oo 



Include on checklist 
Do not include on checklist 
Do mx indude (mi checklist 
Do not include on checklist 



As an iUustration of how lo use the phi cocffidem, consider Item #8 in Figure 2 and apply formuU (I). The matrix is 
as follows; 





Providers 
in Compliance 


Providers 
Out of Compliance 


Row 
Total 


High 
Group 


A = 50 


B ^ 0 


Y = 50 


Low 
Group 


C = 0 


D = 50 


Z = 50 


Column 
Total 


W =50 


X = 50 


Grand 
Total 
100 
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Step A. To calculate the phi coefficient for Item #8, multiply ceil A (SO) by ceU D (50). 
(^)(S0) B 2,500 



Step B. hfext, multiply ceU C (0) by ceU B (0). 
(0)(P) =0 

^ep C SulHnict the answer in Step B fran the aoswa in Step A. 
0500) > (P) a 2^ 

Step D. The next step invt^ves all row ami cdumn totab. Multifdy them together as follows: (W)(X)(Y)(Z). 

(S0)(5OKSO)(S0) o 6^.000 
Sitf E. Take the square roM of Step D. 

Step F. Divide tte amwer in &ep C by tte answer in Sxep E. 
2^ 



2^ 



s -fLOO 



This is the phi coeffictent for Item #8. The resuh in this exam^ imlkates a perfect po^ve correlation between 
the iton and a |m>viiter's being in the high grcHip of i»ovkleTS. This correbtk^ 
table is an excdtem fmSctor. 

OeneraUy, a phi coefficient between (4-^6) and (<f 1.0(9 is indicativ« of a good ^edict^ If thevahieof the 
^ coeffktem is tess then (•!• ^ the item is not a good predictor. If the value is between (•»• .23^ 
too easy. If the vahie is between (0) and (- .2S), tte item is too difficult. If tlwvafaie is between (•»• .26) and (-1.00), 
the item is a terril^ j»edkt(»-. Now ccm^kr a m<m craii^ e»unpte (Item #1 1 in tte fk^^ 
rekvant numbers are shown in the box bekm: 





fas Gofltpfla&cc 


rrOVlsen 

Oat of CotnpHanrr 


Row 
Total 


GrMp 


20 


30 


SO 


Low 
Group 


24 


26 


SO 


Cotame 
Total 


44 


56 


100 



The computational steps are as follows: 
Step A. CeU A times ceU D. (20)(26) = 520 
Step B. CeU B times ceU C. (30)(24) » 720 
Step C Step A minus Step B. (520) - (720) = 



-200 



ERIC 
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step D. CeU W times ceU X times cefl Y times ceU Z. 

(44)(56)(50)(S0) » 6,160,000 
Step E. Square root of Step D. ^6, 160,000 « 2AS2 (rOunded) 
S^F. Step Cdivkled by Step E. » - .08 (rounded) 

This is the phi coefficient for Item '11. The lesuft In thfa dmmpte im^ 

not be used as a predictor item. Ndtto high-group or km-group provides are very successful mcoraplying with the 
regulation that ccmesponds to Iton '11. 
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APPENDIX C: DECIDING THE MONITOR'S RANGE 
OF DISCRETION IN USING THE COMPREHENSIVE 
INSTRUMENT AND THE INDICATOR CHECKLIST 

Suppose a monitor visits a provider to perform a review based on the indicator checklist and then discovers condi- 
ticms that appear to warrant using tht comprehensive iiutrument. Clearly, if the ccmlitions are readily apparent and 
(rtndousiy Imrmfui, ccmimon som will sug^t a simple tkcision: use itw loni^ instrunwnt. In many instances, 
however, the apiHt>priate (tecisi<m wiUbeless dear. For example, should a singh; itm that is out of compliance on the 
ii»licat(»' checklist fot a im>vid^ trigi^ a comprehensive review? Many mmtitcm aiKi state administrates tmy want 
a ckar, formal process for ctedding wlmi ccmditioro warnmt ming the cmiiml^nsive instrument in place of tte in- 
dkatm diecklist. This w^^iix describes semal an;>nMu:l«» that a state may adq;rt. Tte decision of which appr(»ch 
to take will be made by top state administrators based on the state's particular circumstance. 



Approach 1: Permit Monitors Complete 
Dfecretion 

Wtoi a state has sufHcient confktence in the 
judgment of its staff, it may permit mcHutms to use 
timr own discretion in substituting the diecklist for 
the kmgfT imtrunmt For cxampte, at the nuHutor's 
chc^, one itan (Hit of compliaiKe might he enough 
to trigger using the longer instrument, m scvmd 
items out of compliance might insufTicient cause 
to comiuct a more extensive review. 

The apparent drawbacks to this approach arc: 

• potentially reduced confidence in the quality 
of the monitoring effort; 

• irKonsistency in the monitoring effort and 
possible feelings of harassment on the pan 
of some providers who receive comprchen- 

. sive reviews; and 

• reduced savings to the state if monitors 
routii^iy use the comprehensive instrument 
in place of the indicator checklist. 

Nonetheless, the judgment of a knowledgeable, well- 
trained monitor may be a much better guide for 
deciding when to use the longer instrument (or pans 
of it) than the more arbitrary approaches described 
below. 

Approach 2: Single Item Triggers Use of 
Comprehensive Instrument 

A state may determine that the items on the in- 
dicator checklist (whether they are ••essential" items 
or not) are important enough that a single item 
found out of compliance is enough to warrant: 



• a review udng all the itons on the com* 
{Mhensivc instrumott that are in the same 
aroi (e,g., health, safety, transpcmatitm) as 
the item out of cc»npliance; or 

• a comimhensive review of all itans. 

TI^ flrst alternative is more appealing than the 
second, althcHigh eitlm* may be reasonable depetul- 
ing on the teigth and fcHinat of the state's com- 
prrimisive ifutrument« For example, a state may not 
have the items on its long instrument dassirted into 
sections or arras, so that a monito' wcmld have to 
use judgment in selecting which other items to 
review. 

A second potential drawback is that the indicator 
item found out of compliaim may be of low weight, 
and following up with a substantially longer review 
may seem unwarranted. However, the logic that sup- 
pons the selection of items for the checklist also 
provides a good rationale for conducting a com- 
prehensive review if any itffii is found out of ami* 
pliance. That is, non-compliance with any of the 
items indicates a gocKi probability that the provider 
would receive a pror score on (he comprehensive 
instrument. 

Approach 3: Set Threshold Scor^ for 
the Indicator Checklist 

Assuming that items on both the indicator 
checklist and the comprehensive instrument are 
weighted, it is i^ssible to set a ••threshold'* score 
that can be used to decide if a provider should receive 
a comprehensive review. For example, if higher 
scores mean lower compliance, as ts true in Penn- 
sylvania, then a score on the indicator checklist 
above an established cutoff point would 
automatically trigger the use o* tHe comprehensive 
instrument. 
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The threshold score could be set in several ways: 

• Scores on the indicator diecklist could be 
computed for all of tte high-group pro- 
viders whMe soms on tl^ cmnpre te i tt ive 
instnimait were ii^ to sekct checklist 
itrais. Thm iht thre$)M>kl scwt couki be the 
average of these high«group xms. 

• Pillowing the sarm |m>cedure, the scxt of 
the lowe^ ranked imivkier in tte high grcHip 
coukI be used as the thrediold seem. 

• The ^e could arbitrarily sdect a Kon cm 
the indicator checklist to serve as the 
thrediokl score. The m»maU»^ of tte 
scOTc that is selected wouhi deprad on 
policy issues aiui other drtunmaMes in the 
state. 

Though eadi of these alternatives involves a 
policy decision by tte state, the ^vantage in setting a 
thrcslKrfd score b that the state can {novidte an ex- 
pUci! dedsscm rule to guide monitors as they perfcmn 
reviews. This rule <^ both ensure consi^o^ and 
help maintain the quality of the state's monitoring at 
carefully chosen levds. 



